feat: reverse index scans and statement-scoped write visibility - #384
Merged
Merged
Conversation
Executors now hold `&'a LogicalPlan` / `&'a Operator` and copy only the fields they must own, so preparing a plan no longer clones the whole tree per execution. * build_read/build_write take `&'a LogicalPlan`; Input types borrow operators * PlanKeeper keeps an owned plan alive at a stable address for the executor * bind prepared parameters lazily in IndexScan; ParamArena keeps (id, expr) slots and exposes them through MetaArena::bound_param * IndexRanges is a cursor over borrowed ranges (owned only for runtime probes) * SeqScan/IndexScan and DDL executors drop Option+take; DDL and COPY FROM no longer return a result row * populate the output schema for the whole plan once when it is finalized
Bump kite_sql to 0.4.1. The on-disk value format changed (every tuple and index value now carries an 8-byte statement stamp suffix), so data written by 0.4.0 cannot be read. - storage: add `range_rev` to all backends; LMDB/RocksDB iterators are driven by explicit step enums; LMDB `remove_range` reuses the scan. - optimizer/executor: satisfy `ORDER BY ... DESC` / `MAX` with a reverse index scan when the whole order is reversed (NOT NULL columns only, never below a Limit). - fix: scans no longer revisit rows written by the same statement (e.g. primary-key UPDATE, INSERT ... SELECT on the same table, and the same inside explicit transactions). Stamps come from the LMDB txn id, the RocksDB sequence number, or a memory counter, and are only allocated for statements that both scan and write. - memory: iterate the BTreeMap lazily instead of cloning the range. - tests: SLT now runs on LMDB; tpcc runner pins to the fastest P-core.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #384 +/- ##
==========================================
+ Coverage 93.09% 93.26% +0.16%
==========================================
Files 258 259 +1
Lines 47662 47741 +79
==========================================
+ Hits 44370 44524 +154
+ Misses 3292 3217 -75
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue link:
ORDER BY ... DESC/MAXalways needed a full index scan plus sort/TopK; TPC-C Order-Status got slower as orders grew.UPDATE t SET id = id + 1000 WHERE id > 3updated rows multiple times (or until integer overflow), andINSERT INTO t SELECT ... FROM tduplicated rows. RocksDB had the same bug inside explicit transactions.What is changed and how it works?
&LogicalPlanoperators instead of copying them; prepared statements no longer clone the plan. Parameters are bound lazily inIndexScan.range_revon all storages; the optimizer reverses an index scan when the whole required order is reversed (NOT NULL columns only, never below aLimit).EXPLAINshowsReverse.INSERT ... VALUESuses 0).remove_rangereuses the scan; memory iterates the BTreeMap lazily.Code changes
Check List
Tests
Side effects
Note for reviewer
main; Order-Status p90 about -15% (51→43 µs). With the stamp check disabled the result is the same, so the cost comes from elsewhere in the change, most likely the extra 8 bytes per value (not profiled). RocksDB Delivery p90 dropped about 34% in a separate short run.update.slt/insert.sltand a shared explicit-transaction test on memory, LMDB, and RocksDB (pessimistic + optimistic) fail without the fix.unsafe:PlanKeeper, the memory iterator value pointer,mdb_txn_id, and the RocksDB snapshot sequence number, each with aSAFETYcomment. Not run under Miri.Range::Eqhits) are not filtered by stamp; table-levelPRIMARY KEY (a, b)does not mark columns NOT NULL, so reverse scans don't apply there.